I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Extract object-level and column-level data flow from optimized query plans through Plan::query_lineage(), and expose stable column IDs in system.columns for downstream lineage resolution.

The extraction result is storage-independent: this PR does not persist lineage or add a query interface. #20230 consumes this planner model through the history-table pipeline.

Motivation

Lineage consumers need a canonical representation of which input relations and columns contribute data to each target relation and column. Reconstructing those edges later from formatted SQL would duplicate binder semantics and would be unreliable for aliases, casts, subqueries, CTEs, optimizer rewrites, mutation expressions, Views, and Streams.

The planner already has resolved relation identities, column IDs, scalar expressions, and the optimized relational tree. Extracting lineage there produces a reusable model while leaving persistence and query policy to downstream consumers.

Stable column IDs are also exposed through system.columns. This lets a consumer resolve an ID-addressed historical column to its current name after a rename, while correctly rejecting a dropped-and-recreated column that reuses the old name with a new ID.

Output model

QueryLineage records the statement kind and canonical data-flow edges:

source relation/column -> target relation/column

The model groups source relations under each target relation. Column edges are sorted and deduplicated. Relations carry catalog, database, name, optional stable ID, catalog type, and relation kind.

Supported target statements include:

• CREATE TABLE ... AS SELECT (CTAS)
• CREATE VIEW when the caller supplies its optional query plan
• INSERT ... SELECT and INSERT from Stage
• REPLACE INTO ... SELECT
• multi-table INSERT
• COPY INTO
• mutation plans used by UPDATE ... FROM and MERGE

Statements without a data-producing query return no lineage.

Resolution algorithm

Extraction has two phases:

1. Walk the optimized SExpr and build symbol definitions for base scans and data-producing operators.
2. Starting from each target value expression, recursively resolve referenced symbols until base relation columns are reached.

Definitions cover scalar evaluation, aggregate functions, window functions, set-returning functions, UDFs, async functions, UNION ALL, expression scans, scalar subqueries, and automatic materialized CTE references.

Only target value expressions are traversal roots. Columns used exclusively by filters, join predicates, WHEN conditions, or ordering do not become lineage unless they also contribute to an output expression. Aggregate arguments such as count(col) propagate col, while count(*) has no input column edge.

Cycle detection prevents recursive symbol definitions from looping, and ordered sets keep multi-source expressions deterministic.

Relation semantics

• View reads stop at the View output boundary instead of expanding into its defining base tables.
• Stream data columns transparently resolve to the backing table; Stream metadata columns are ignored.
• Automatic materialized CTE consumer symbols map back to producer symbols.
• Explicit AS MATERIALIZED CTE lineage is not included because the query-scoped temporary relation does not retain its original producer-column mapping.
• Named Stage scans produce object-level sources but no file-field column mapping.
• Temporary tables, MEMORY, DELTA, and unsupported sources are skipped.
• Default-catalog relations retain stable IDs. Hive, Iceberg, and Paimon relations use their catalog identity and omit unstable runtime table IDs.
• Self-loop candidates remain in the planner model because accurate filtering requires resolved source and target object identity; persistence consumers can remove them.

For ordinary INSERT, the target ID is stored only in a lineage field. table_info remains unset, so execution still resolves the target by catalog/database/table name and is not pinned to the object observed during planning.

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Planner coverage includes CTAS, Views, INSERT SELECT, View and Stream boundaries, Stage scans, CTEs, automatic materialized CTEs, aggregates, joins, subqueries, REPLACE, UPDATE, MERGE, self-insert candidates, external catalog addressing, and multi-table INSERT.

The sqllogic test verifies that system.columns exposes stable column IDs.

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

AI assistance

• AI usage: An AI coding agent assisted with planner extraction, regression tests, commit organization, local validation, and drafting this PR description; the responsible human directed the design and reviewed the final behavior and diff.
• Responsible human: @youngsofun
• The responsible human has read every line of this diff and can explain each change

This change is